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We study how the degree of symmetry in the couplings influences the performance of 
three mean field methods used for solving the direct and inverse problems for generalized 
Sherrington-Kirkpatrick models. In this context, the direct problem is predicting the po- 
tentially time- varying magnetizations. The three theories include the first and second order 
Plcfka expansions, referred to as naive mean field (nMF) and TAP, respectively, and a mean 
fleld theory which is exact for fully asymmetric couplings. We call the last of these simply 
MF theory. We show that for the direct problem, nMF performs worse than the other two 
approximations, TAP outperforms MF when the coupling matrix is nearly symmetric, while 
MF works better when it is strongly asymmetric. For the inverse problem, MF performs better 
than both TAP and nMF, although an ad hoc adjustment of TAP can make it comparable 
to MF. For high temperatures the performance of TAP and MF approach each other. 
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1. Introduction 

Predicting the dynamical properties of a disordered system given a specific reali- 
sation of its parameters is an old and important problem in statistical mechanics. 
This is what one can call a direct problem. Apart from being important on its own, 
solving the direct problem is also a crucial step in solving the inverse problem: in- 
ferring the parameters of a system from measurements of its dynamics. With the 
rapid advance of methods for observing the dynamics of biological systems com- 
posed of many elements, the inverse problem has received a lot of recent attention. 
This line of research has allowed inferring functional and physical connections in 
neuronal networks [1-5], gene regulatory networks [6] and protein residue contacts 
[7]. 

A useful platform for studying the inverse problem is a dynamical version of the 
Sherrington-Kirkpatrick (SK) model: a set of N classical spins, Sj = ±1 subject 
to a potentially time- varying external field hi{t) with couplings Jij between them 
and a stochastic update rule. In the direct problem one tries to predict the magne- 
tizations mi{t) given the coupling and fields. In the inverse problem one does the 
opposite, i.e. one infers the couplings and the fields from measured magnetizations 
and correlations. 

When the system is in equilibrium and the distribution of states follows the 
Boltzmann distribution, several approaches for both direct and inverse problems 
have been developed. These include both exact and approximate iterative algo- 
rithms, such as Boltzmann learning and Susceptibility propagation [8, 9] relating 
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the magnetizations to model parameters, as well as closed-form equations based on 
naive mean field (nMF) and TAP [10, 11] equations for the SK model. Motivated 
by the fact that biological systems are usually out of equilibrium, some recent 
work has focused on reconstructing the parameters of a dynamical Ising spin glass 
model obeying either synchronous or asynchronous updating from observing its 
out-of-equilibrium dynamics [5, 12, 13]. 

In this paper, we investigate how three recently proposed mean field methods 
for the direct and inverse problems perform on models with different degrees of 
symmetry in their coupling matrices. The three methods are the nMF and TAP 
equations, derived using the high-temperature Plefka expansions of the generating 
functional to first order and second order [14], and a mean field theory (denoted 
simply MF) [13] that is exact for the SK model with fully asymmetric couplings. 



2. Solutions to the direct and inverse problems 

We consider a model in which the probability of being in state s at time step t, 
Pt{s), is given by 



Pt+i{s) = Y,Wt[s;s']pt{s') (la) 



j 

For the choice of couplings Jjj, we follow [15], taking 

j^^ = jiy-^ + kJt;y-^ (2) 

where J-j = J^ " is the symmetric part oi the couplings while J- = —Jji is 
the antiymmetric part. All the couplings J^^"^ and J^j^y"^ are drawn independently 
from a zero-mean Gaussian distribution with variance 

r jsymmi2 _ TJasymi2 _ 3^ ('y\ 

ya \ -^ij J - (1 + ^2)^- w 

With Eqs. 2 and 3, the couplings Jij have variance of /N and the degree of 
symmetry is controlled by k: for = the model is fully symmetric {Jij = Jji) 
while for k = 1, it is fully asymmetric {Jij independent of Jji). 

The direct problem consists in estimating the instantaneous magnetization of 
spin i at time t, mi{t). The estimation obtained from the nMF, TAP and MF are 
respectively: 

mi{t + 1) = tanh hi{t) + Jijmj{t) (4a) 
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mi{t + 1) = tanh hi{t) + J] Jijmj{t) - mi{t + l)Y^ Jfj{l - m]{t)) 



(4b) 



mi{t + l) 



dx 



27r 



"^'/^ tanh \hi{t) + Jijirijit) + x^Ai{t)\ (4c) 



where in the last equation 



(5) 



For deriving Eqs. 4a and 4b, i.e. nMF and TAP, one first writes down the gener- 
ating functional for the process defined by Eq. 1, performs a Legendre transform to 
fix the magnetizations and expands the results for small g (i.e. high temperature). 
To the first order, this expansion gives the nMF equations, Eq. 4a. Keeping terms 
up to the second order yields a correction to the nMF equations resulting in the the 
TAP equations, Eq. 4b, for this dynamical model. nMF and TAP are, therefore, 
high temperature expansions for an arbitrary set of couplings, with no assumption 
about their distribution or its degree of symmetry. The third equation is derived 
for arbitrary g, but under the mean-field assumption that at each time step the 
fields acting on the spins are independent Gaussian variables. This is exact for this 
SK model when the coupling matrix is fully asymmetric i.e. when A; = 1. 

These direct equations can also be used for solving the inverse problem. The 
idea is to use the data in order to measure the magnetizations m,i{f), the equal 
time correlations Cjj = {6si{t)6sj(t)), and the time-delayed correlations Dij = 
{Ssi{t + l)Ssj{t)), where Ssi{t) = Si{t) — mi{t). For the process in Eq. 1, one can 
write the time-delayed correlations as 



Dij = (tanh [ei{t)\sj{t)) - (tanh [0i(i)])(sj(t)). 



(6) 



To derive the inverse TAP and nMF, one then uses Eq. 6, expands the tanh around 
mi that satisfies one of the direct equations 4a and 4b. In the case of MF, one writes 
an expression for the joint distribution of Oi{t) and 9j{t) that is exact for a fully 
asymmetric SK model. This joint distribution can then be used to relate JD to C 
in the limit of small Cij] for details see [5, 13]. Within all three approximations, 
nMF, TAP, and MF, the resulting expression takes the form 



D = AJC , 



(7) 



where the matrix yl is a diagonal matrix that depends on the approximation: 



A, 



nMF 
ij 



5ij{l-m1) , 



(8a) 



4AP = <5,,(l-m2)(l-F,), 



(8b) 



dx 



1 - tanh^ {hi{t) + Y Jij mj + x y/Ki) . (8c) 
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In Eq. 8b Fi satisfies a cubic equation. For details see [5] and [13]. Not surprisingly, 
expanding Eq. 8c to linear or second order in Jij yields A^^^ and A^^^ in Eqs. 8a 
and 8b, respectively. 

Eq. 7 can be solved for J = A^^DC^^, provided one has enough data so that 
the estimation of C is good, allowing its numerical inversion. 

3. Effect of Symmetry 

As mentioned before, for the direct problem, we expect that the MF becomes exact 
for = 1 for any coupling strength g. TAP equations should also become exact 
for A; = in the limit of weak couplings. This is shown in Fig. 1, where we plot 
the mean squared error in predicting the magnetizations at time t + 1 given the 
magnetizations at time t. This is done both for a constant field and for an external 
field that varies sinusoidally with time. As can be seen in this figure, for both types 
of external fields, TAP equations outperform the other two methods for small k. 
As temperature is increased, all three approximations perform better and become 
almost equally good. As k increases, MF wins over TAP while nMF performs worse 
than both of them. 

The situation for the inverse problem is slightly more complicated. This is be- 
cause, for strong couplings, the cubic equation that Fi solves develops complex 
roots. In this case one can take three approaches: (i) take the nMF result, (ii) take 
the real part of the solution, (iii) take the solution for the largest g for which the 
solutions are real. This value can be shown to be Fi = 1/3. The results for the 
last two strategies almost coincide, with strategy (iii) performing slightly better in 
lower temperatures, so we chose this one. In strategy (i) the results just coincide 
with the nMF approach after the temperature at which the cubic equation for F^ 
develops complex roots. The results from strategy (iii) are shown in Fig. 2. It is 
clear from this figure that nMF always performs worse than the other two and 
that the difference between the three methods vanishes in the high-temperature 
limit. On the other hand, MF is superior, as expected, when one gets closer to the 
asymmetric case i.e. for k is close to 1. The TAP result has a more complicated 
behavior, due to the intrinsic limitations imposed by the lack of real solutions of 
the cubic equation at strong couplings. However, one can notice that, when k is 
close to zero, there is a range of couplings g where TAP becomes better than MF 
as it is expected. 

As can be seen in the right column of Fig. 2, the mean squared error 
^ jmfered _ jr?a')2 becomes larger for non-zero external fields. This is a general fea- 
ture of all three methods. Large fields and/or couplings are estimated with larger 
errors than small ones. This is because errors in the estimation of the empirical 
magnetizations/correlations, when the later are close to ±1, produce large errors 
in the estimation of the fields/couplings (consider for example, in zeroth order 
approximation, a sigmoid map between m,j and hi , and Cij and Jij). Numerical 
simulations show that, for large external field amplitude, these errors become so 
important that the differences between the three methods are insignificant. 

4. Conclusions 

Within the mean field approaches that we have studied, the solution of the inverse 
problem derives from the solution of the direct problem. We have studied here 
three methods that provide an approximate solution to the direct problem in the 
case of systems with infinite range interactions. We have explored their behaviors 
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on both the direct and the inverse problem in the case of SK models with different 
degrees of symmetry of the interactions. As expected, the MF approach is the best 
one when the degree of asymmetry is large enough, but the TAP approach turns 
out to be slightly better in some range of coupling strength when the couplings are 
more symmetric. The nMF approach is just a first order approximation to both 
MF and TAP and is systematically worse than the other two methods. 

As noted before, the derivation of inverse nMF and TAP rely on expanding the 
tanh in the around the solutions of the nMF and TAP. This expansion is not 
required for the MF solution: in the case with the assumption of full asymmetry, 
the joint distribution of the local field to each pair of spins will be Gaussian and 
can be easily calculated. It is therefore possible to write an exact equation relating 
Dij to Cij and the couplings which in the limit of small Cij can be linearized and 
takes the form of Eq. 7. It would be interesting to see if a similar approach can be 
done within the TAP framework: calculate the joint distribution of the local fields 
in a systematic small coupling expansion, and use the same procedure done in MF 
to relate Dij to Cij. 

In real applications, for instance in neural data analysis or gene regulation net- 
work reconstruction, one docs not deal with data generated from a model with the 
particular size dependence of the couplings of the SK model. Our previous work 
shows that TAP and nMF perform at the same level in identifying the connections 
of a simulated neural network, and they both perform worse than the exact iter- 
ative Boltzmann like learning rule that one can write down for the dynamical SK 
model [5, 16]. We will leave the comparison of TAP, MF and the exact learning on 
biological data to future work. 
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Figure 1. 

LEFT PANELS: Mean squared error of the three methods for predicting the magnetizations at time t given 

at time t — I, averaged over spins and times, (m^'^°'^"^*°'^ (t) — m™'^^'^"''"'* (t))^ . This mean squared error is 
plotted as a function of g for a system of size A'^ = 50 with a temporally constant field drawn independently 
for each spin from a normal distribution. We have used 100 time steps and 50000 repeats to calculate 
the experimental magnetizations and have averaged the errors over 10 realizations of the couplings. The 
different figures correspond to different values of k. From top to bottom k = 0,0.25,0.5,0.75,1. RIGHT 
PANELS: The same but with the addition of a sinusoidal external field of period 10 time steps and 
amplitude 0.5. 
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Figure 2. 

LEFT PANELS: Mean squarre error of the three methods on the infered couphngs (J'"*°'^'^'* — J^J'^')^ as 
a function of g for systems of size N = 100 with zero external field, given P = 100000 patterns, averaged 
over 10 realizations of the couplings. The different figures correspond to different values of k. From top to 
bottom k = 0,0.25,0.5, 0.75, 1. RIGHT PANELS: The same but with the addition of a sinusoidal external 
field of period 10 time steps and amplitude 0.5. 



